Shiny for R-Package Risk Assessment


Aaron Clark, 16 Aug 2023

Disclaimer




Any opinions expressed in this presentation and on the following slides are solely those of the presenter and do not necessarily reflect those sponsoring the work

Agenda

  • Quick Intro to {riskmetric} / {riskassessment}

  • Why create a shiny app?

  • Latest enhancements

  • Demo

  • Coming soon

  • Q&A

  • Consortia of ~50 companies (mostly pharma and biotech)

  • Focus on designing/building a framework to validate R and R packages with an eye to all open-source software.

  • The main goal is ensuring proper validation documentation exists in accordance with expectations of regulatory agencies.

Checkout pharmar.org to learn about all work streams and get involved!

Two tools: what do they do



is a framework to quantify an R package’s “risk” by assessing several meaningful metrics designed to evaluate package development best practices, code documentation, community engagement, and development sustainability.


is a full-fledged R package containing a shiny front-end that augments the utility of {riskmetric}. The application’s goal is to provide a central hub for an organization to review and assess the risk of R packages, providing handy tools and guide rails along the way.

Quantify risk programmatically

Sometimes “quality” is measurable! Software dev best practices dictate an R-package should have:

  • A license
  • Source code available for browsing
  • An easy to contact maintainer
  • A place to report bugs
  • Evidence that new bugs are being addressed
  • Complete Function documentation
  • Adequate test coverage
  • Community usage

18 total assessments (to date)!

Why create a Shiny app?


Main goal: help those those making “package inclusion” requests for GxP environments. So, the app’s highest and best use is all about two things:

  • Forcing org members to take responsibility for assessing package risk themselves prior to making an IT request
  • Generating a summary report for IT that shows the requester has done their due diligence meeting inclusion requirements

Why create a Shiny app? (cont’d)

Main goal: help those those making “package inclusion” requests for GxP environments. They need support to…

  • Provide a platform for package exploration without the need to write any custom {riskmetric}
  • Run {riskmetric} on the same machine with the same environment – creating a central hub for reproducibility
  • Maintain consistent, org-specific settings/options when producing risk outputs
  • Automate a risk-based “decision triage” based on pre-defined rules, saving time & effort
  • Manage who’s involved in the review process via user authentication & role management
  • Facilitate and store user summaries & communication, on certain packages and/or certain metrics
  • Generate risk summary reports, for sharing with the decision making parties

Latest features (v2.0.0)

Most notably:

  • Face lift to the 'Report Builder' & 'Database View'
  • Better support to analyze dependencies
  • More org-level customization, including the use of a configuration file
  • Allow admin users to edit roles and privileges on the fly
  • Allow users to explore source contents of package


The feedback loop is crucial! Submit an issue on GitHub today.

Latest: 'Report Builder' Face Lift

  • A more holistic Report Builder! Allowing users to define what content shows up in the report.

  • Users can now compose a long form “Package Summary” to keep track of more pertinent items (perhaps non-{riskmetric} items) for a more rounded package review.

Latest: 'Database View' Face Lift

More useful content to ingest!

  • A summary of uploaded packages
  • Date uploaded
  • Decision-related columns like the decision time stamp & decision source
  • Easily downloadable

Latest: Package Dependencies

Latest: org-level settings

Edit in-app or via the config file! What’s new?

  • Customize decision categories & colors
  • Toggle decision automation rules
  • User roles/privileges
  • Initialize metric weights

Example config (YML) file:

default:
  assessment_db: database.sqlite
  loggit_json: loggit.json
  credential_db: credentials.sqlite
  credentials:
    roles:
      - admin
      - lead
      - reviewer
      - viewer
    privileges:
      admin: [admin, weight_adjust, auto_decision_adjust, final_decision, revert_decision, add_package, delete_package, overall_comment, general_comment]
      lead: [weight_adjust, auto_decision_adjust, final_decision, revert_decision, add_package, delete_package, overall_comment, general_comment]
      reviewer: [add_package, general_comment]
      viewer:
  decisions:
    categories:
      - Low Risk
      - Medium Risk
      - High Risk
    rules:
      Medium Risk:
        - .398
        - .639
      High Risk:
        - .639
        - 1
    colors:
      Medium Risk: !expr grDevices::rgb(52, 235, 229, maxColorValue = 255)
  metric_weights:
    covr_coverage: 0
    has_vignettes: 2



Automated a risk-based “decision triage” based on pre-defined rules, saving time & effort

Decisions impacted upon pkg upload:

Latest: Edit Roles & Privileges

Manage who’s involved in the review process via user authentication & role management

Latest: Explore Package Source Code

Augment {riskmetric} evaluation with a more manual package review process by browsing source code

Latest features recap (v2.0.0)

Most notably:

  • Face lift to the 'Report Builder' & 'Database View'
  • Better support to analyze dependencies
  • More org-level customization, including the use of a configuration file
  • Allow admin users to edit roles and privileges on the fly
  • Allow users to explore source contents of package


The feedback loop is crucial! Submit an issue on GitHub today.

An exercise: Review the {prodlim} package


Demo on Shinyapps.io!


Test drive now: bit.ly/raa_demo

Coming soon

{riskscore}

In it’s early stages, a new data package containing the {riskmetric} assessments & scores for all packages on CRAN.

Coming soon

Package Explorer

A suite of hands-on review tools (from our friends at GSK) will allow users to browse unit tests, source code, and man (doc) pages for any exported function.

Dev Team

Q & A

bit.ly/raa_demo

bit.ly/raa_gh

bit.ly/raa_survey

pharmar.org/